Evaluating methods for computer-assisted stemmatology using artificial benchmark data sets

Given a collection of imperfect copies of a textual document, the aim of stemmatology is to reconstruct the history of the text, indicating for each variant the sourcc tcxt from it was copied. We describe an experiment involving three artificial bcnchmark data sets to which a number of computer-assisted stemmatologr mcthods wcre applied. Contrary to earlicr similar expcrimenls, we propose and use a numerical criterion to evaluate all the solutions. Moreover, our primary data set is significantly !arger than used before. The results suggest the superiorit}' of two computer-assisted methods amongst those tested: the maximum parsimony method implemented in the PAUP• software package and a related compression-based method we have proposed in earlier work. 'No book is published without some discrepancy in each one of the copics. intentional modifications. They accumulated in copics of copies, copics of copies of copies, ctc. Consequently, a text of any importance ended up existing in a group of different vcrsions, a so called tmdition, some of which are all but identical to the original, some perhaps hardly recogni7.able. Connccting each version to its exemplar, i.e. the version from which it was copied, gives a tree-like structure called the stcmma, with the original version as the root. The aim of stemmatolog}' is to recover this structure given a set of surviving variants. Scribes tal-e a secret oath to omit, to intcrpolate, to change.' (Jorgc Luis Borges: Tile LoNery in Bab}•lc111, in Labyrinths: Seleded Stories & Otlrcr Writings,